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INTRODUCTION:  In  this  project,  we  proposed  to  research  the  methodology  for  constructing 
adaptive  computer-aided  education  systems  for  mammography.  The  project  includes:  Observer 
studies  to  collect  reading  data  from  radiology  trainees;  Extraction  of  image  features  (human- 
and  computer-  based)  from  mammograms;  Statistical  modeling  of  trainees  reading  data  to 
discover  patterns  in  their  error  making;  Development  of  methodology  for  adaptive  training  that 
utilizes  the  constructed  models.  The  proposed  adaptive  system  could  improve  education  in 
mammography.  This  may  in  turn  result  in  improved  benefit  of  mammography  in  breast  cancer 
detection  and  lower  mortality  associated  the  disease. 

BODY: 

Overall  progress: 


Specific  aim 

Expected 

Actual 

1.1  Prepare  the  database  of  screening  mammograms 
(year  1,  months  1-6) 

Completed 

Completed  in 
respect  to  the 
data  for  the  first 
reader  study 

1.2  Obtain  the  approval  for  the  human  subject 
(observer)  studies  in  tasks  1  and  3. 

Completed 

Completed 

1.3  Perform  an  observer  study  in  which  residents  will 
search  for  masses  and  architectural  distortions 
(year  1,  months  7-9).  We  expect  20  human  subjects 
(observers)  to  participate  in  this  study. 

Completed 

Completed 

1.4  Utilize  the  user  data  collected  in  the  observer  study 
to  develop  machine  learning-based  individual  user 
models  (year  1,  month  10  -  year  2,  month  6) 

In  progress 

In  progress 

The  detailed  description  of  progress  regarding  each  specific  aim  follows. 


1.1  Prepare  the  database  of  screening  mammograms  (year  1,  months  1-6) 

STATUS:  completed  in  respect  to  the  data  for  the  first  reader  study 
DETAILS: 

In  order  to  conduct  a  reader  study  we  had  to  collect  a  set  of  mammographic  cases  including 
normal  and  abnormal  cases.  This  step  included  securing  access  to  Duke's  clinical  data  for  the  PI. 
In  order  to  retrieve  cases,  we  identified  a  set  of  potential  patients  and  exams  using  Duke's 
informatics  system  and  then  analyzed  the  patient's  data,  including  radiology  reports,  to  find 
exams  that  could  be  used  in  this  study.  Then,  images  were  retrieved  using  PACS  system.  One 
hundred  digital  mammographic  cases  were  used  for  the  first  reader  study. 

DIFFICULTIES: 

The  mammographic  cases  were  originally  proposed  to  be  retrieved  from  the  database  of 
screen-film  mammograms  (DDSM)  available  online.  This  original  plan  was  changed.  Since  we 
believe  that  using  digital  mammograms  instead  of  screen-film  mammograms  is  more  relevant 


to  the  current  and  future  clinical  practice,  we  decided  to  use  digital  mammograms  for  this 
study.  The  change  was  discussed  with  a  DOD  program  official.  This  radically  increased  the  time 
of  the  data  collection  since  instead  of  simply  downloading  the  mammograms  from  an  already 
organized  online  database,  we  had  to  go  through  an  extremely  time  consuming  retrospective 
data  collection  process  at  Duke.  Additionally,  since  the  collection  of  the  images  involved  human 
subjects,  we  had  to  wait  for  the  IRB  approval  to  start  collecting  the  data.  While  the  data 
collection  time  was  increased,  we  were  able  to  collect  a  usable  set  of  digital  mammography 
cases  for  the  first  reader  study.  Additional  mammograms  will  be  collected  as  needed. 

1.2  Obtain  the  approval  for  the  human  subject  (observer)  studies  in  tasks  1  and  3. 

STATUS:  completed 
DETAILS: 

We  obtained  an  approval  for  the  human  subject  studies.  Specifically  the  approval  was  secured 
for  (1)  retrospective  collection  of  mammograms,  and  (2)  collection  of  data  in  reader  (observer) 
studies.  The  approval  was  first  secured  within  Duke  and  then  the  protocol  was  approved  by  the 
DOD.  Following  the  approval,  multiple  amendments  were  introduced  to  the  protocol  in  the 
course  of  this  project. 

DIFFICULTIES: 

Since  we  decided  to  include  digital  mammograms  collected  at  Duke  rather  than  originally 
proposed  online  database  DDSM  (this  change  was  discussed  with  a  program  official),  the 
protocol  had  to  include  details  on  the  retrospective  collection  of  images.  This  difficulty  was 
resolved. 

1.3  Perform  an  observer  study  in  which  residents  will  search  for  masses  and  architectural 
distortions  (year  1,  months  7-9).  We  expect  20  human  subjects  (observers)  to  participate 
in  this  study. 

STATUS:  completed 

DETAILS: 

For  this  study  a  graphical  user  interface  (GUI)  was  developed.  The  GUI  was  developed  in  the  C# 
programming  language  using  the  Microsoft  Visual  Studio  software.  The  GUI  allowed  for  viewing 
and  marking  the  mammograms  as  well  as  answering  the  questions  related  to  each  case.  The 
GUI  software  saved  the  reading  data  for  future  analysis. 

Monitor  and  a  computer  workstation  were  also  purchased  using  the  grant  funds. 

Subjects  for  the  reader  study  were  recruited  from  Duke  and  UNC  Chapel  Hill.  A  total  of  18 
subjects  participated  in  the  study.  Three  of  them  were  breast  imaging  attendings  (as  reference 
data).  Out  of  remaining  15  subjects,  10  provided  data  that  we  were  able  to  use  in  our 
experiments.  The  reasons  for  a  subject's  data  not  being  usable  included  misunderstanding  of 
the  instructions.  Out  of  the  10  usable  subjects,  7  were  residents,  2  were  researchers  and  1  was 
a  medical  student.  The  non-resident  participants  were  included  to  resemble  pre-residency 
trainees. 


Figure  1  The  workstation  and  the  graphical  user  interface  prepared  and  used  for  the  reader  study 

DIFFICULTIES: 

As  the  study  was  time  consuming  (~3  hours),  it  has  proven  difficult  to  recruit  subjects.  However, 
though  various  recruitment  efforts  at  two  medical  centers  (Duke  and  UNC  Chapel  Hill)  we  were 
able  to  recruit  a  reasonable  number  of  subjects  that  we  can  use  in  our  analysis. 

1.4  Utilize  the  user  data  collected  in  the  observer  study  to  develop  machine  learning-based 
individual  user  models  (year  1,  month  10  -  year  2,  month  6) 


STATUS:  in  progress 
DETAILS: 

This  task  consists  of  developing  a  set  of  human-extracted  and  computer-extracted  image 
features  as  well  as  utilizing  these  features  to  build  models  that  predict  how  the  trainees  make 
errors.  We  initiated  research  in  this  direction. 

Specifically,  we  conducted  a  preliminary  analysis  for  some  of  the  data  collected  in  the  reader 
studies.  Since  the  focus  of  our  study  is  on  finding  cases  that  are  difficult  for  each  trainee,  it  is 
important  to  better  understand  the  concept  of  case  difficulty.  We  performed  analysis  that 
allows  for  answering  questions  regarding  the  relationship  between  self-assessed  difficulty 
("subjective"  assessment  of  difficulty)  and  actual  error  made  for  cases  ("objective"  assessment 


of  difficulty).  The  analysis  was  submitted  and  accepted  to  SPIE  Medical  Imaging  2013 
conference  and  will  be  published  next  year  (upon  submitting  the  final  version  of  the 
proceedings  manuscript).  We  will  attach  the  final  conference  proceedings  paper  when  it  is 
published.  The  preliminary  results  showed  that  there  is  a  notable  relationship  between  the 
subjective  assessment  of  difficulty,  and  error  (e.g.  true  positive  rate  and  false  positive  rate). 
More  detailed  statistical  analysis  is  pending. 

Furthermore,  we  are  working  on  constructing  a  controlled  lexicon  for  extraction  of  imaging 
features  from  mammograms  by  radiologists.  The  controlled  lexicon  is  likely  to  decrease  inter¬ 
observer  variability  for  the  features.  I  am  collaborating  with  two  breast  imaging  faculty  at  Duke 
and  one  at  the  UNC  Chapel  Hill  to  obtain  a  comprehensive  set  of  mammographic  descriptors 
that  might  relate  to  error  in  residents. 

We  also  initiated  the  computerized  image  analysis  step  as  well  as  investigated  other,  related 
ways  of  user  modeling  toward  reaching  the  goals  of  this  study.  These  analyses  are  in  progress. 

DIFFICULTIES: 

No  major  difficulties  have  been  encountered  as  of  this  point. 


KEY  RESEARCH  ACCOMPLISHMENTS: 

•  Secured  IRB  approval  for  the  study 

•  Retrospectively  collected  a  set  of  mammograms  for  the  study 

•  Conducted  an  observer  study 

•  Conducted  preliminary  analysis  of  the  observer  study  results 

•  Initiated  development  of  a  controlled  dictionary  for  mammography  education 


REPORTABLE  OUTCOMES: 

•  Collected  a  database  of  digital  mammograms 

•  An  extended  abstract  entitled  "Difficulty  of  mammographic  cases  in  the  context  of 
resident  training:  preliminary  experimental  data"  submitted  and  accepted  to  SPIE 
Medical  Imaging  2013  conference. 

CONCLUSION: 

The  work  proceeds  mostly  according  to  the  original  timeline.  Since  in  the  first  year,  we 
proposed  to  mostly  focus  on  preparation  to  research  (IRB  approval),  collection  of  images  and 
collection  of  reader  data,  no  major  scientific  discoveries  are  reported  yet.  However,  thorough 
execution  of  these  first  steps  is  crucial  for  the  success  of  the  entire  project. 


APPENDICES:  No  appendices  at  this  time 


